Psychometric characteristics of integrated multi- specialty examinations: Ebel ratings and unidimensionality

Over recent years UK medical schools have moved to more integrated summative examinations. This paper analyses data from the written assessment of undergraduate medical students to investigate two key psychometric aspects of this type of high stakes assessment. Firstly, the strength of the relationship between examiner predictions of item performance (as required under the Ebel standard setting method employed) and actual item performance (‘facility’) in the examination is explored. It is found that there is a systematic pattern of difference these two measures, with examiners tending to under-estimate the difficulty of items classified as relatively easy, and over-estimating that of items classified harder. The implications of these differences for standard setting are considered. Secondly, the integration of the assessment raises the question as to whether the student total score in the exam can provide a single meaningful measure of student performance across a broad range of medical specialties. Therefore Rasch 1 Corresponding author. Leeds Institute of Medical Education, School of Medicine, University of Leeds, Leeds LS2 9JT